Augmented reality procedural system

ABSTRACT

An augmented reality device is provided to assist users in performing new or unfamiliar experimental techniques, identify materials and products utilized in a documented action set and within a work environment, identify equipment and instruments needed in the documented action set and within the work environment, assist in performing single person (autonomous) work, collaborate with other workers, and record data and observations in an electronic laboratory notebook.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority and benefit under 35 U.S.C. 119 to U.S.application Ser. No. 62/265,255, filed on Dec. 9, 2015, titled“AUGMENTED REALITY PROCEDURAL SYSTEM”, which is incorporated byreference herein in its entirety.

BACKGROUND

Material handling generally, and in laboratories specifically, canbenefit from greater uniformity, consistency, reliability,reproducibility, quality, and integrity. Laboratory protocols orlaboratory notebooks may improve some of these issues, but reference aprotocol manual or lab notebook for an experimental protocol orprocedure can be time consuming and tedious. In addition, some protocolsand procedures may require measurement value and results to be reportedafter each measurement or at intervals that may overlap with thepreparation of a subsequent step of the action set.

Identification and material mixing is a common part of everyday life,and even more so in the life of a scientist. Determining the outcomes ofvarious interactions of known and unknown substances is a challengingproblem to overcome. Further, the physical properties calculated bycurrent physics engines are generally restricted to physics, and fail totake into account the effect of additional science fields on materials.Controlling computer interfaces and peripheral machines based on theseinteractions creates another level of complexity.

The following references provide background on the material disclosedherein: Eppela, S. and Kachman, T. (2014) Computer vision-basedrecognition of liquid surfaces and phase boundaries in transparentvessels, with emphasis on chemistry applications. arXiv preprintarXiv:1404.7174

Fraczek J, Zlobecki, A., and Zemanek, J. (2007) Assessment of angle ofrepose of granular plant material using computer image analysis. Journalof Food Engineering 83, 17-22

OpenSim. (2016) Referencing OpenSim Arm26 model and OpenSim itselfhttp://simtkconfluence.stanford.edu:8080/display/OpenSim/About+OpenSim,andhttp://simtk-confluence.stanford.edu:8080/display/OpenSim/Induced+Acceleration+Analysis,download 18 Nov. 2016.

RecFusion (2016). Software by imFusion GmbH, Munich, ImFusion GmbH,Agnes-Pockels-Bogen 1 80992 Munchen, Germany. Downloaded 18 Nov. 2016from www.recfusion.net.

Vassiliou, E., Giunto, G. F., Schaefer, W. R., Kahn, B. R. (1994). Slagviscosity detection through image analysis of dripping slag withinrotary incineration kilns. U.S. Pat. No. 5,301,621.

BRIEF SUMMARY

Disclosed herein are augmented reality (AR) systems and procedures toenable AR device such as the Microsoft® Hololens® to facilitate materialhandling activities. The systems and procedures may be applied tochemistry, biology, construction, manufacturing, cooking, and many otheractivities that involve material handling and which benefit from beingreproducible and teachable.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

To easily identify the discussion of any particular element or act, themost significant digit or digits in a reference number refer to thefigure number in which that element is first introduced.

FIG. 1 illustrates an aspect of an augmented reality measurement system100 including a computer system 116 in accordance with one embodiment.

FIG. 2 illustrates another aspect of the augmented reality measurementsystem 100 in accordance with one embodiment.

FIG. 3 illustrates an augmented reality measurement system 300 inaccordance with one embodiment.

FIG. 4 illustrates an augmented reality measurement system 400 inaccordance with one embodiment.

FIG. 5 illustrates an augmented reality measurement system 500 inaccordance with one embodiment.

FIG. 6 illustrates the computer system 116 communicatively coupled to anetwork 602.

FIG. 7 illustrates an augmented reality process 700 in accordance withone embodiment.

FIG. 8 illustrates a subroutine block 708 in accordance with oneembodiment.

FIG. 9 illustrates an embodiment of a system 900 for interacting withsubstances.

FIG. 10 illustrates an embodiment of a process 1000 for interacting withsubstances.

FIG. 11 illustrates an aspect of the subject matter in accordance withone embodiment.

FIG. 12 illustrates an embodiment of a procedure 1200 for guiding anddocumenting procedural actions of an action set through an augmentedreality device.

FIG. 13 is an example block diagram of a computing device 1300 that mayincorporate embodiments of the present invention.

DETAILED DESCRIPTION

References to “one embodiment” or “an embodiment” do not necessarilyrefer to the same embodiment, although they may. Unless the contextclearly requires otherwise, throughout the description and the claims,the words “comprise,” “comprising,” and the like are to be construed inan inclusive sense as opposed to an exclusive or exhaustive sense; thatis to say, in the sense of “including, but not limited to.” Words usingthe singular or plural number also include the plural or singular numberrespectively, unless expressly limited to a single one or multiple ones.Additionally, the words “herein,” “above,” “below” and words of similarimport, when used in this application, refer to this application as awhole and not to any particular portions of this application. When theclaims use the word “or” in reference to a list of two or more items,that word covers all of the following interpretations of the word: anyof the items in the list, all of the items in the list and anycombination of the items in the list, unless expressly limited to one orthe other. Any terms not expressly defined herein have theirconventional meaning as commonly understood by those having skill in therelevant art(s).

The terminology used herein has its conventional meaning unlessotherwise defined below:

“Augmented reality” in this context refers to is a live direct orindirect view of a physical, real-world environment whose elements areaugmented (or supplemented) by computer-generated sensory input such assound, video, graphics or GPS data, in real time, to enhance a user'scurrent perception of reality. Augmented reality herein also refers topurely virtual environments influenced by objects or actions in thephysical world (i.e., “virtual reality”).

“Augmented virtuality” in this context refers to the dynamic integrationof physical (real) world objects or actions to interact with a virtualenvironment.

“Comparative Physical Modeling” in this context refers to a logicalcomponent that identifies similarities in the computational model of adetected object's physical attributes with existing models of knownmaterials under the same measurement conditions.

“Environmental Signal” in this context refers to a complex of sensorysignals originating from a plurality of sensors measuring physicalattributes of a protocol environment.

“Graphical Overlay” in this context refers to a transparent visual layeron a display partially obscured by graphical objects.

“Mixed Reality” in this context refers to the merging of real andvirtual world objects and interactions to produce new environments andvisualisations where physical and digital objects co-exist and interactin real time.

“Particle modeler” in this context refers to logic to perform particlebehavior modeling. Open source particle modeling logic includes LIGGGTSand E-Sys Particle. Particle modelers build a computational model of agiven particle system based on observed attributes of the particlesystem and known (previously stored) attributes of substances.

“Quantifying Features” in this context refers to structures or regionsof a laboratory instrument visually inspected to obtain measurements orvalues.

“Spatiotemporal Movement Analysis” in this context refers to a logiccomponent that derives physical attributes to generate a computationalmodel for a detected object in the protocol environment.

“User Space” in this context refers to a voided region occupiable by auser operating an augmented reality device.

“Vector modeler” in this context refers to any logic to model 2D or 3Dprocesses or motion using vectors. Conventional systems providing vectormodeling logic include QGIS open source vector modeler library, and theLive2D modeler from Live2D Corp. Vector modelers construct vectors forthe mechanics of motion for an object or particle. This may involvereading and calculating force, velocity, acceleration and other aspectsof kinematics, statics and dynamics dealing with the motion of objects.

“Visual display” in this context refers to a display of an augmentedreality device that shows a graphical overlay. The visual display mayadditionally include a representative video layer of the graphicalenvironment. Alternatively, the visual display may be transparent inconstruction.

“Action multiplexer” in this context refers to a logic component thatselects one of several possible analog or digital input signals toinitiate an action on a machine by forwarding the selected action inputto an output terminal.

“Predictive model” in this context refers to known and observedattributes of substances which are combined to build a data model whichenables predictions about future interaction based on the combinedattributes of the component substances.

“Sensor” in this context refers to a device or composition of matterthat responds to a physical stimulus (as heat, light, sound, pressure,magnetism, or a particular motion) and transmits a resulting impulse (asfor measurement or operating a control).

“Associator” in this context refers to a correlator (see the definitionfor Correlator).

“Circuitry” in this context refers to electrical circuitry having atleast one discrete electrical circuit, electrical circuitry having atleast one integrated circuit, electrical circuitry having at least oneapplication specific integrated circuit, circuitry forming a generalpurpose computing device configured by a computer program (e.g., ageneral purpose computer configured by a computer program which at leastpartially carries out processes or devices described herein, or amicroprocessor configured by a computer program which at least partiallycarries out processes or devices described herein), circuitry forming amemory device (e.g., forms of random access memory), or circuitryforming a communications device (e.g., a modem, communications switch,or optical-electrical equipment).

“Classifier” in this context refers to a specific type ofcorrelator/associator logic that associates one or more inputs with acategory, class, or other group sharing one or more commoncharacteristics. An example of a classifier that may commonly beimplemented in programmable hardware is a packet classifier used innetwork switches, firewalls, and routers (e.g., packet classifiersutilizing Ternary Content Addressable Memories). An example software orfirmware classifier is: if (input1.value<12.5) input1.group=group1; elseif (input1.value>=12.5 and input1.value<98.1) input1.group=group2; elseinput1.group=group3; Other examples of classifiers will be readilyapparent to those of skill in the art, without undo experimentation.

“Combiner” in this context refers to a logic element that combines twoor more inputs into fewer (often a single) output. Example hardwarecombiners are arithmetic units (adders, multipliers, etc.),time-division multiplexers, and analog or digital modulators (these mayalso be implemented is software or firmware). Another type of combinerbuilds an association table or structure (e.g., a data structureinstance having members set to the input values) in memory for itsinputs. For example: val1, val2, val3→combiner logic→{val1, val2, val3}set.val1=val1; set.val2=val2; set.val3=val3; Other examples of combinerswill be evident to those of skill in the art without undoexperimentation.

“Comparator” in this context refers to a logic element that compares twoor more inputs to produce one or more outputs that reflects similarityor difference of the inputs. An example of a hardware comparator is anoperational amplifier that outputs a signal indicating whether one inputis greater, less than, or about equal to the other. An example softwareor firmware comparator is: if (input1==input2) output=val1; else if(input1>input2) output=val2; else output=val3; Many other examples ofcomparators will be evident to those of skill in the art, without undoexperimentation.

“Correlator” in this context refers to a logic element that identifies aconfigured association between its inputs. One examples of a correlatoris a lookup table (LUT) configured in software or firmware. Correlatorsmay be implemented as relational databases. An example LUT correlatoris: |low_alarm_condition |low_threshold_value|0∥(safe_condition|safe_lower_bound|safe_upper_bound∥high_alarm_condition|high_threshold_value|0|Generally, a correlator receives two or more inputs and produces anoutput indicative of a mutual relationship or connection between theinputs. Examples of correlators that do not use LUTs include any of abroad class of statistical correlators that identify dependence betweeninput variables, often the extent to which two input variables have alinear relationship with each other. One commonly used statisticalcorrelator is one that computes Pearson's product-moment coefficient fortwo input variables (e.g., two digital or analog input signals). Otherwell-known correlators compute a distance correlation, Spearman's rankcorrelation, a randomized dependence correlation, and Kendall's rankcorrelation. Many other examples of correlators will be evident to thoseof skill in the art, without undo experimentation.

“Firmware” in this context refers to software logic embodied asprocessor-executable instructions stored in read-only memories or media.

“Hardware” in this context refers to logic embodied as analog or digitalcircuitry.

“incrementer” in this context refers to logic to advance (increase ordecrease) a counting or index value by a fixed or predictably variableamount. Examples of hardware incrementers include adder arithmeticcircuits and counter circuits. An example of a software incrementer is:x=x+incrementValue. Incrementers may be used as counters, or as logic toadvance a referencial or associative index in a memory data structure.

“Logic” in this context refers to machine memory circuits, nontransitory machine readable media, and/or circuitry which by way of itsmaterial and/or material-energy configuration comprises control and/orprocedural signals, and/or settings and values (such as resistance,impedance, capacitance, inductance, current/voltage ratings, etc.), thatmay be applied to influence the operation of a device. Magnetic media,electronic circuits, electrical and optical memory (both volatile andnonvolatile), and firmware are examples of logic. Logic specificallyexcludes pure signals or software per se (however does not excludemachine memories comprising software and thereby forming configurationsof matter).

“Parser” in this context refers to logic that divides an amalgamatedinput sequence or structure into multiple individual elements. Examplehardware parsers are packet header parsers in network routers andswitches. An example software or firmware parser is:aFields=split(“val1, val2, val3”, “,”); Another example of a software orfirmware parser is: readFromSensor gpsCoordinate; x_pos=gpsCoordinate.x;y_pos=gpsCoordinate.y; z_pos=gpsCoordinate.z; Other examples of parserswill be readily apparent to those of skill in the art, without undoexperimentation.

“Programmable device” in this context refers to an integrated circuitdesigned to be configured and/or reconfigured after manufacturing. Theterm “programmable processor” is another name for a programmable deviceherein. Programmable devices may include programmable processors, suchas field programmable gate arrays (FPGAs), configurable hardware logic(CHL), and/or any other type programmable devices. Configuration of theprogrammable device is generally specified using a computer code or datasuch as a hardware description language (HDL), such as for exampleVerilog, VHDL, or the like. A programmable device may include an arrayof programmable logic blocks and a hierarchy of reconfigurableinterconnects that allow the programmable logic blocks to be coupled toeach other according to the descriptions in the HDL code. Each of theprogrammable logic blocks may be configured to perform complexcombinational functions, or merely simple logic gates, such as AND, andXOR logic blocks. In most FPGAs, logic blocks also include memoryelements, which may be simple latches, flip-flops, hereinafter alsoreferred to as “flops,” or more complex blocks of memory. Depending onthe length of the interconnections between different logic blocks,signals may arrive at input terminals of the logic blocks at differenttimes.

“Selector” in this context refers to a logic element that selects one oftwo or more inputs to its output as determined by one or more selectioncontrols. Examples of hardware selectors are multiplexers anddemultiplexers. An example software or firmware selector is: if(selection_control==true) output=input1; else output=input2; Many otherexamples of selectors will be evident to those of skill in the art,without undo experimentation.

“Sequencer” in this context refers to logic to generate an ordered listof outputs from either an unordered or partially ordered set of inputs,or from a starting input and rules to generate next inputs. Oneattribute of a sequencer is that the outputs are done sequentially,meaning one after the other in time. An example of a hardware sequenceris a multiplexer with a counter driving its selection input. An exampleof a software or firmware sequencer is: out=val++; Other examples ofhardware and software or firmware sequencers will now be readilyapparent to those of skill in the relevant arts.

“Software” in this context refers to logic implemented asprocessor-executable instructions in a machine memory (e.g. read/writevolatile or nonvolatile memory or media).

“Switch” in this context refers to logic to select one or more inputs toone or more outputs under control of one or more selection signals.Examples of hardware switches are mechanical electrical switches forswitching power to circuits, devices (e.g., lighting), or motors. Otherexamples of hardware switches are solid-state switches such astransistors. An example of a hardware or firmware switch is: if(selection==true) output=input; else output=0; A somewhat morecomplicated software/firmware switch is: if (selection1==true andselection2==true) output=input 1; else if (selection1==true andselection2==false) output=input2; else if (selection1==false andselection2==true) output=input3; else output=noOp; Switches operatesimilarly to selectors in many ways (see the definition of Selector),except in some cases switches may select all inputs to the output,(s)not select among inputs. Other examples of switches will be readilyapparent to those having skill in the art, without undo experimentation.

An augmented reality device is provided to assist users in performingnew or unfamiliar experimental techniques, identify materials andproducts utilized in a documented action set and within a workenvironment, identify equipment and instruments needed in the documentedaction set and within the work environment, assist in performing singleperson (autonomous) work, collaborate with other workers, and recorddata and observations in an electronic laboratory notebook.

A method of guiding and documenting procedural actions of an action setmay be provided with an augmented reality device, operatively disposedto a user space, comprising a processor, memory, a plurality of externaltransducers, a plurality of internal transducers, and a visual display,and provided with a graphical overlay superimposed on an ocular field ofview represented on the visual display.

An environmental signal, captured through a plurality of externaltransducers, may be transformed into an image map including detectedobjects. Equipment and instruments may be recognized from the detectedobjects through comparative analysis of visual characteristics of thedetected objects with known visual characteristics in an equipment andinstrument database.

Materials and products may be identified from the detected objectsthrough spatiotemporal movement analysis, comparative physical modeling,and referencing a materials database. The detected objects may besynchronized to the graphical overlay to coincidently align the detectedobjects of the image map with the environmental objects of theinteraction environment to the ocular field of view through operationsof the processor controlled by overlay synchronization logic.

Objects of interest within the image map are recognized by transformingan ocular movement tracking signal, captured by a plurality of internaltransducers, into an ocular line of sight directed towards anenvironmental object in the interaction environment, with acorresponding detected object in the image map. Procedural actions arerecognized through contextual visual behavior recognition by trackingspatiotemporal movement of the materials and products relative to theequipment and instruments in the series of sequential image maps, andcorrelating to documented procedural actions in an action set databaseas controlled by behavior identification logic.

The augmented reality measurement system may provide active guidance forperforming lab procedures, including visual indicators identifying anext action to take as an overlay on material or equipment from theocular field of view displayed on the image map. The augmented realitymeasurement system may also identify and visually indicate proceduralerrors when the user's actions do not comport with a next action totake.

Quantifiable data may be captured by quantifying features of theequipment and instruments through contextual feature analysis, and bycorrelating the procedural action and the ocular line of sight to thequantifying features of the equipment and instruments in the series ofsequential image maps, and visualizing a guided interaction overlay asthe graphical overlay in response to an action set selection input of adocumented action set from the action set database. The guidedinteraction overlay may include directional indicators such as arrows,dots, lines, and text (for example) indicating equipment or materialswith which to interact, operating or material handling instructions,and/or next actions to take.

An action set outline may be visualized detailing the documentedprocedural actions of the documented action set through the guidedinteraction overlay. A list of materials and a list of equipment andinstruments utilized in the documented action set may also be visualizedthrough the guided interaction overlay. A graphical identifier alignedto the detected objects may be displayed in the image map correspondingto materials, equipment, and instruments in the interaction environmentthrough the guided interaction overlay. One or more advisementnotifications may be displayed through the guided interaction overlay inresponse to detecting an advisement and precaution alert associated withparticular documented procedural actions.

In some embodiments, the augmented reality device includes a userinterface to receive vocalized inputs. The vocalized inputs may beequipment and instrument identifiers, material and product identifiers,interaction environment identifiers, or an action set modificationnarrative.

In the aforementioned embodiment, equipment and instrument identifiersand material and product identifiers may be recognized from thevocalized input as controlled by speech recognition logic. The equipmentand instrument identifiers and the material and product identifiers maybe assigned to particular objects of interest as controlled byuser-object identification logic. Analysis metrics for the behavioridentification logic and the object recognition logic may be adjusted ascontrolled by environment-dependent identification logic. An action setmodification narrative may be recognized, and in response a modifiedaction set outline may be stored as an interaction entry in an actionlog allocation of memory as controlled by the action set documentationlogic.

Capturing the quantifiable data may include identifying character valuesin the quantifying features of the equipment and instruments throughoperations of the processor under control of optical characterrecognition logic, and storing the character values, equipment andinstrument identifiers, material and product identifiers andspatiotemporal identifiers as an interaction entry in an action logallocation of memory as controlled by action set documentation logic.

The quantifiable data, within the interaction entries stored in theaction log allocation of memory, may be accessible through a graphicaluser interface. The graphical user interface may display thequantifiable data mapped to the procedural steps as well as any modifiedaction set outline that occurred during the execution of the action set.

A user may establish particular attributes for the interactionenvironment and for objects within the interaction environment through auser interface. The user interface may accept direct inputs such as textinput and vocalized input to establish identification of a particularinteraction environment or a particular object. Through theidentification of the particular interaction environment and/or theparticular object, the system retrieves established attributes andprioritizes results from the materials database, the equipment andinstrument database, and the action set database based on one or more ofprevious interactions within the particular interaction environment,particular object, previous interactions in protocol environments withsimilar attributes to the particular interaction environment, or objectswith similar attributes to the particular interaction environment. Inone example, the user interface may accept textual input identifying anexternal network database or website detailing attributes of aparticular object or particular procedures of an action set.

The augmented reality device is typically communicably coupled to anetworked repository that may include the materials database, theequipment and instrument database, and the action set database. Theaugmented reality device may transmit the environmental signal to adistribution server for distribution to collaborators. The recognitionof the materials and products within the series of sequential image mapsmay be performed as an asynchronous process performed by the processorunder control of a subroutine of the object recognition logic.

A plurality of internal transducers (internal to the augmented realitydevice) may be used to collect a plurality of biometric/physiologicalreadings from a user within the user space. The bio-metric readings mayinclude, but are not limited to, heart rate data, motion data,respiratory data, temperature data, cardiovascular data, andelectrophysiological data.

The augmented reality device may be autonomous with on-board autonomousoperational logic and data repositories, but is more likely operatedusing a combination of onboard and external logic and data repositories.Moreover, the augmented reality device may communicate with otherresearchers and devices worn by and carried by them. The augmentedreality device may respond to input and requests from the researcher viacues generated verbally, sub-vocally, by eye movements and othergestures, by head movements, by specific body movement, by learnedunconscious body movements, by learned autonomic cues, and combinationsof these. These inputs may be received directly or relayed from otherdevices (e.g. from accelerometers or other sensors on smartphones orsmartwatches) on the person of the user or remote from the user (e.g.cameras).

The device may also accept input that requires interactive communicationbetween device and user, in which the user, for example, responds to arequest that they mechanically manipulate a container holding somematerial, from which data the interactive interfaced augmented realitymeasurement system makes inferences about the material's mass anddensity. The user may be taught the identities of objects and materialsand given additional didactic or useful information about objects in theuser's visual field. The interfaced augmented reality measurement systemmay recognize, identify, and tag laboratory objects in the user's fieldof view. On request from the user, the augmented reality device mayprovide additional information (e.g. who the object belongs to,manufacturer of object, vendor, the last time object is known to havebeen touched or moved). The augmented reality device may requestadditional input from the user such as information about mass anddensity of samples inferred from imaging the user hefting a containerincluding the sample.

The user may be taught to perform specific experimental techniques,often, by interaction with online information including informationresident in laboratory protocols. In this aspect, the augmented realitymeasurement system responds to input from the user by identifyingspecific objects that are the objects of the user's intendedmanipulation. For example, a user may indicate using voice cues thatthey want to streak out a bacterial culture onto an agar dish to isolatesingle colonies. The interactive interface augmented reality device asksthe user to examine the lab space, and locate a test tube containing abacterial culture, a Bunsen burner, a wire loop, and a petri dish withsterile agar. The augmented reality device may show the user text,images and/or video of the procedure, and offer voice, text, or imageguidance to the next steps. The user may focus visual attention on someaspect of the procedure at any point. The device may offer the userinput as they perform an action on the focused aspect. For example,should the visual image of the procedure deviate from the image theaugmented reality device associates with correct performance of theprocedure, the augmented reality device may inform the user of thediscrepancy. Similarly, the user may stop moving at some point andrequest comparison with images or video (photographic or animationrepresentation) of the idealized procedure.

The user may be aided by the augmented reality device while theyfunction autonomously in a research lab (i.e. carry out research). Theaugmented reality measurement system may recognize and accept input onobjects in the visual field, for example as described above. Theaugmented reality device may interact with the user to increase theuser's knowledge of expected behaviors that arise while carrying outexperiments. The augmented reality measurement system may have or mayinterface with an Augmented/Mixed Reality Physics Model (AMRPM, as anAugmented/Mixed Reality Laboratory Physics Model, AMRLPM) executinglogic (called an Augmented/Mixed Reality Physics Engine, AMRPE, as anAugmented/Mixed Reality Laboratory Physics Engine, AMRLPE) that embodiesexpectations about behavior of objects, equipment, and substances.Examples include the transfer of heat to liquids in a flask with a stirbar perched on a hotplate, the rate at which different solid compoundsdissolve in aqueous buffers, the rate at which different hyrogscopicpowdered solids might take on moisture from the air, estimated volumesand masses of solids and liquids from visual data or prompted hefts, thebehaviors of liquids and bubbles in tubing and microfluidic devices,etc. The augmented reality measurement system may enable theinvestigator to compare these expected behaviors to what they areexperiencing. The AMRLPM may also aid in training and followingprocedures as described previously.

The augmented reality measurement system may aid the recording andinterpretation of observations. For example, the user may be examining afield of view such as colonies on a bacterial plate, or Arabdopsisplants in a pot. The augmented reality measurement system may identifythe visualized objects, recording their number, sizes, and otherrelevant visual data. The system may then count, catalog and recorddifferent categories of objects in the visual field. The system maydisplay this information to the user for their approval. The user may,by interaction with the device, train the interface in more accurateidentification, and/or identification that is more consonant with thecriteria articulated by the researcher or taught by the researcher, tothe AI. This information may be communicated to a research group, viasocial media, to an archive, etc. The system submit data recorded forresearch and training via interaction with a human teacher, or for crowdsourced evaluation via entities such as Stack Exchange.

A user of the augmented reality measurement system may workcollaboratively with students and researchers, to conduct training orresearch. The system may enable all of the above uses and collaborativeworkers may be able to exchange fields of view containing viewed objectsand other entities in field of view, may designate objects of commoninterest, may request input about the field of view and the objects init, and may participate in other activities, for example interacting togenerate consensus determinations about data gathered and recorded.

Referring to FIG. 1, an augmented reality measurement system 100includes a camera 102, a human operator 106 outfitted with augmentedreality headset 108, and a container 110 holding a liquid. The augmentedreality measurement system 100 may be used to measure characteristics ofa liquid sample in the container 110.

The container 110 is separated from the human operator 106 by a sampledistance 104. Within the container 110, the liquid has discernableliquid boundaries 112. 3) The container 110 may be a known type, forexample a flask or graduated cylinder, of known volume, for example, atwo liter Erlenmeyer flask of standard geometry.

U.S. Pat. No. 5,301,621, by Vassiliou et al. discloses image analysistechniques for measurement of properties of drippings and isincorporated herein by reference in its entirety.

The container 110 is imaged from at least two perspectives, one from theaugmented reality headset 108 and one from the camera 102. Other camerasmay be utilized to provide additional fields of view on the container110. Referring to FIG. 2, the container 110 is then oriented such thatdroplets 202 emerge from it.

Liquid from the container 110 is poured at a constant rate slow enoughto produce the droplets 202. This may entail feedback from the augmentedreality headset 108 to the human operator 106 to modify the speed(faster or slower) that the droplets 202 are poured out, and may forexample utilize a display on the augmented reality headset 108 of anangle 204 to orient the container 110 to produce the desired rate ofpour, based on real-time analysis of whether or not the droplet profilenecessary for viscosity and/or density analysis is being formed.

The type of the container 110 may be determined by way of computerizedobject recognition using the camera 102 and/or augmented reality headset108, or because the human operator 106 designates the type of thecontainer 110 using voice recognition. Dimensions of the container 110are then either calculated from the image of the container 110 (e.g., ascompared to a size in the image of a reference object in the field ofview, the reference object at a known distance from the camera 102 oraugmented reality headset 108). Dimensions of the container 110 may alsobe extracted from an equipment and instrument database 114 thatassociates object types with their dimensions. The augmented realityheadset 108 may communicate wirelessly with a computer system 116 via awireless gateway 118 to read and write equipment data to the equipmentand instrument database 114 and from a sample database 120 comprising anassociation of material properties with known compositions.

The sample distance 104 may be determined by calibration to a knowndistance (e.g., the length of an arm of the human operator 106) and/orby utilizing a depth sensor system of the augmented reality headset 108.The camera 102 may be used together with the augmented reality headset108 to triangulate the sample distance 104. The augmented realityheadset 108 may also incorporate sonic depth sensors (e.g., LIDAR).Distance determination algorithms utilizing image analysis and/or sonicsignals are known in the art.

For the measured angle 204, a size of each of the droplets 202 and anumber of the droplets 202 falling per second through a defined verticaldistance 206 are measured. For a defined axis of symmetry 208 aroundwhich the droplets 202 can be assumed to be symmetrical, a count ofimage pixels per droplet may provide a working analytical basis forestimating the size and volume of the droplets 202.

An aspect ratio of the droplets 202 (e.g., how round vs. teardrop shapethey are) may also be determined from a pixel count and the axis ofsymmetry 208 for each droplet.

An estimate of the density and/or viscosity of the droplets 202 may bedetermined by measuring the fall time for a volume, aspect ratio, andcross section (e.g., widest cross section) as each drop falls throughthe vertical distance 206. For many lab experiments, the verticaldistance 206 may vary between 0.2 meter and one meter of air at standardEarth gravity acceleration and atmospheric pressure. The determineddensity for many droplets 202 may then be averaged to determine anapproximate density of the liquid in the container 110.

The density and/or viscosity of the droplets 202, and/or their aspectratio and cross section may also be utilized to quality the compositionof the liquid in the container 110, by referencing the sample database120 for known compositions having those properties. Other featuresrecorded by the camera 102 and/or the augmented reality headset 108 thatmay be utilized for the qualitative analysis of the composition includeit's color and light permeability (opaqueness).

A volume of liquid in the container 110 may be determined by firstdetermining a volume of the container 110 as previously described (e.g.,identifying the type of the container 110 and looking up its volume inthe equipment and instrument database 114, or via image analysis). Theliquid boundaries 112 may then be identified using known techniques. Onesuch technique is described in “Computer vision-based recognition ofliquid surfaces and phase boundaries in transparent vessels, withemphasis on chemistry applications” by Sagi Eppel and Tal Kachman,Cornell University Library, 6 Nov. 2014.

The volume of liquid in the container 110 may then be estimated from theliquid boundaries 112 and using values interpolated from numbers readfrom the container 110, and/or values stored in the equipment andinstrument database 114, or using volumetric equations specific to thegeometry of the container 110.

Referring to FIG. 3, an augmented reality measurement system 300 may beutilized to determine the composition, density, and mass of materialfrom image analysis of a container 302 being hefted, poked, squeezed,jiggled, shaken, or otherwise physically perturbed.

The human operator 106 applies approximately calibrated forces to thecontainer 302. Calibration and control of the applied forces may befacilitated by the augmented reality headset 108, which may incorporatefeedback from a force feedback sensor 304 in a glove or other device.

The augmented reality headset 108 and/or camera 102 may capture andanalyze images or video of the disturbance or change to material in thecontainer 302 (e.g., movement, acceleration, deformation), for exampleas disclosed by Güler et al. 2014 and Güler et al. 2015, below.

Güler, P., Bekiroglu, Y. Gratal, X. Pauwels, K. and Kragic, D. “What'sin the container? Classifying object contents from vision and touch”,2014 IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS 2014). DOI: 10.1109/IROS.2014.6943119

Güler, P., Pauwels, K. Pieropan, A., Kjellstr{umlaut over ( )}om H., andKragic, D. (2015). “Estimating the deformability of elastic materialsusing optical flow and position-based dynamics”, IEEE-RAS InternationalConference on Humanoid Robots, 2015. DOI: 10.1109/HUMANOIDS.2015.7363486

The augmented reality measurement system 300 may for example utilizelookup tables in the sample database 120 cross referenced to lookuptables in the equipment and instrument database 114 for records ofexperimentation (e.g., hefting, poking, squeezing, jiggling, andshaking) containers of known volume filled with substances of knowncomposition, including information about composition, density and mass.The tables may comprise empirical results of actions on containersperformed by a set of people who span a gamut of known genders andsizes, and levels of physical fitness (eg. 50 kilogram women, 90kilogram men), or by calibration using objects of known properties bythe specific human operator 106.

The augmented reality measurement system 300 may in some embodimentsestimate masses and forces being exerted to cause quantified motion byreference to simulated force generated by human arms of known dimensionsmoving masses of known weight, using for example open source softwareand models such as OpenSim (for example the Arm26 Model, 2016).

Referring to FIG. 4, an augmented reality measurement system 400 may beused to determine characteristics of a powdered or granulated sample404. In one embodiment characteristics of the powdered or granulatedsample 404 may be determined from an angle of repose 402 of the powderedor granulated sample 404 using inputs from the augmented reality headset108 and one or more strategically positioned camera 102.

One type of characterization for the powdered or granulated sample 404is the determination of amount of moisture or size of the grains in thepowdered or granulated sample 404. This determination may be facilitatedby a measure of the angle of repose 402 of the powdered or granulatedsample 404.

One or more image from the augmented reality headset 108 and/or thecamera 102 may be analyzed to calculate the angle of repose 402, forexample as described in conjunction with FIG. 4 of “Assessment of angleof repose of granular plant material using computer image analysis”, byJ. Fraczek et al., Journal of Food Engineering 83 (2007) 17-22.

The angle of repose 402 may also be utilized to determine a volume ofthe powdered or granulated sample 404, according to a base radius orbase diameter 406 of the powdered or granulated sample 404 calculated byscale analysis of an image of the powdered or granulated sample 404, andthe sample vertical distance 408 from the camera 102 (or the augmentedreality headset 108, or both).

To identify the portion of the image that includes the powdered orgranulated sample 404, the human operator 106 may point manually to thepile, or the augmented reality headset 108 and/or camera 102 mayidentify the most pile-like object in the image through analysis or bymatching against an image library of powdered or granulated samples, forexample as stored by the sample database 120.

The human operator 106 may delineate the powdered or granulated sample404 manually using hand gestures to trace the boundaries of the powderedor granulated sample 404 or to crop the image sufficiently that analgorithm such as described in Fraczek et al. is able to accuratelyoperate.

If the powdered or granulated sample 404 has a shallow angle of repose402, that is indicative that the powdered or granulated sample 404 hasnot gained a great deal of moisture from the environment. If the angleof repose 402 is steep, that is indicative that the powdered orgranulated sample 404 has gained significant moisture content. The angleof repose 402 for a particular type of powdered or granulated sample404, for example Argo cornstarch, may be recorded as a function ofmoisture in the sample database 120.

The particle size of a powdered or granulated sample 404 of knowncomposition and moisture content may also be determined from the angleof repose. Larger particles will tend to have shallower angle of repose402 for a fixed moisture composition. By first assuming a substantiallydry (very low moisture content) for the powdered or granulated sample404 of known composition, e.g., for flour), the particle diameter may beestimated by measuring the angle of repose 402 and comparing it to drysample values for the powdered or granulated sample 404 in the sampledatabase 120.

FIG. 5 illustrates another process to determine the amount of moistureor size of the grains in the powdered or granulated sample 404, referredto herein as a “tilting box” process. The powdered or granulated sample404 is positioned within a container 502 with one or more transparentside to observe the behavior of the powdered or granulated sample 404and the angle of repose 402 under mechanical change. The base diameter406 of the material is leveled and parallel to the base of the container502. The container 502 is slowly tilted at a rate, for example, of 0.3°per second. Tilting of the container 502 is stopped when the powdered orgranulated sample 404 begins to slide in bulk, and the angle 504 of thetilt is measured using the augmented reality headset 108 and/or camera102. The angle 504 may then be compared to angles associated with sampletypes having different material compositions, at different moisturelevels, in the sample database 120 to determine a material and/ormoisture composition of the powdered or granulated sample 404.

FIG. 2 illustrates an augmented reality measurement system 600 in whichthe computer system 116 is connected to one or more server 604 (e.g., aLAN server or web server) via a network 602. The server 604 providesaccess to a reference library 606 repository database. Contents of thereference library 606 may be indexed and searchable using a searchengine 610.

In various embodiments, network 602 may include the Internet, a localarea network (“LAN”), a wide area network (“WAN”), and/or other datanetwork. In addition to traditional data-networking protocols, in someembodiments, data may be communicated according to protocols and/orstandards including near field communication (“NFC”), Bluetooth,power-line communication (“PLC”), and the like. In some embodiments,network 602 may also include a voice network that conveys not only voicecommunications, but also non-voice data such as Short Message Service(“SMS”) messages, as well as data communicated via various cellular datacommunication protocols, and the like.

In various embodiments, the computer system 116 may include desktop PCs,mobile phones, laptops, tablets, wearable computers, or other computingdevices that are capable of connecting to network 602 and communicatingwith server 604, such as described herein.

In various embodiments, additional infrastructure (e.g., short messageservice centers, cell sites, routers, gateways, firewalls, and thelike), as well as additional devices may be present. Further, in someembodiments, the functions described as being provided by some or all ofserver 604 and computer system 116 may be implemented via variouscombinations of physical and/or logical devices.

The reference library 606 may include digital materials (copyrighted orotherwise), for example digital reference material licensed toscientists, experimenters, teachers, and/or students from universitylibraries. The digital materials may include laboratory proceduresgenerated and stored using embodiments of the augmented realitymeasurement systems described herein. By sensing and interpretinggestures, voice, or other user commands, an augmented realitymeasurement system may utilize the computer system 116 to record,retrieve, and display digital documents or lab procedures to and fromthe reference library 606, either directly or through links provided bythe search engine 610. Retrieved materials or links thereto may bepresented to the user of an augmented reality device on a visual displaymerged with objects, equipment, and/or materials in represented in theocular field of view, or rendered to the user audibly, or saved in theuser's electronic laboratory notebook, or stored in personal (to thehuman operator 106) or institutional data repositories.

In some implementations of the system 600, the augmented reality headset108 or the computer system 116 may include a usage meter 608 to tracktime of display and/or interaction between the human operator 106 anddigital materials from the reference library 606. Usage informationcaptured by the usage meter 608 may be communicated back to thereference library 606 for analysis.

The augmented reality measurement system detects measurable propertiesof materials in a user space and assists with identifying the type andstate of the materials in question. The augmented reality measurementsystem may predict and report to the human operator 106 interactionsbetween substances on a particle-level. Measuring and observingproperties such as optical density, birefringence, schlieren, viscosity,and surface properties of a liquid, allows for inferences to be madeabout the that liquid, such as the number and size of cells and otherparticles in the liquid. Changes in optical density and above propertiesmay also be indicative of changes in state in cells in culture (e.g.,changes in size, shape, and partial lysis). When used to controlperipheral machines, this becomes a particularly powerful technologywhich allows for a greater degree of precision in machine control ofmaterial handling by the human operator 106.

The outputs generated by the augmented reality measurement system mayupdate an augmented reality or mixed-reality interface of the augmentedreality headset 108 or other augmented reality device, enabling thehuman operator 106 to track material behavior predictions output by thesystem. The human operator 106 may adapt their actions or, if necessary,take manual control of the equipment and instruments in the ocular fieldof view if the predicted outcome is not congruent with the expected ordesired outcome of the material handling process. The augmented realitymeasurement system may employ machine learning to improve its model andpredictions of materials, equipment, and instruments, recording resultsand feedback from the human operator 106 and refining predictions forfuture interactions.

Referring to FIG. 7, an augmented reality process 700 for assisting ahuman operator 106 with execution of stored material handling proceduresmay be carried out as follows:

-   -   begin executing guided material handling procedure 702;    -   set initial FOV object enhancements 704;    -   perform stored procedure lookahead 706; a recorded and stored        description of a material handling procedure from a memory of        the augmented reality headset 108 or the computer system 116, or        an external system (e.g., reference library 606), may be        analyzed for interactions with equipment or instruments in the        user space that are more imminently relevant;    -   increase enhancements for FOV objects having increasing temporal        relevance 710; this may include enhancing or adding an augmented        reality outline of the more relevant objects, warming (changing        toward a higher thermal temperature color, such as red or yellow        or white) their overlayed or represented augmented reality        color, or other visual enhancement techniques known in the art;    -   decrease enhancements for FOV objects having decreasing temporal        relevance 712; this may include de-enhancing or removing an        augmented reality outline of the more relevant objects, cooling        (changing toward a lower thermal temperature color, such as red        or yellow or white) their overlayed or represented augmented        reality color, or other visual de-enhancement techniques known        in the art;    -   is the procedure done? 714; if yes, the process 700 concludes;        otherwise, it loops back to perform stored procedure lookahead        706 again.

The warming and cooling of objects in the graphical overlay may beproportional to their distance (in number of actions, or in time, or inprobability of being interacted with given the current position in thedocumented action set) from a current position in the documented actionset being facilitated by the augmented reality measurement system. Inone embodiment, warming and cooling and gradual processes (e.g.,gradient color changes) and not abrupt transitions in the representationof the equipment and instruments corresponding to upcoming actions inthe documented action set in the graphical overlay.

The augmented reality process 700 continually monitors a storedprocedure that is actively being carried out by the human operator 106,identifies a current point in the stored procedure at which the humanoperator 106 is operating, and looks ahead in the stored procedure forupcoming events involving interactions with equipment or instruments inthe user space, and applies gradual augmented reality enhancements tothe appearance of more temporally relevant (more relevant to upcomingactions in the procedure) objects. The enhancements are visuallyproportional to the imminence of the object's use in the storedprocedure.

Referring to FIG. 8, a subroutine block 708 to perform stored procedurelookahead 706 may be carried out as follows:

-   -   identify the current action being carried out in the stored        procedure being carried out by the human operator 802;    -   identify in the stored procedure a set of upcoming (subsequent)        actions to the current action 804; and    -   determine the time intervals between current action and upcoming        actions 806.

Enhancements (e.g., temperature color adjustments) to equipment andinstruments in the ocular field of view may then be set based on thesize of the time intervals (e.g., shorter intervals correspond to warmercolor settings for the objects).

Referring to FIG. 9, a system 900 for interacting with substancesincludes sensor 904, sensor 906, multiplexer 908, image processor 902,action multiplexer 920, data model 924, substance 922, processor 918,vector modeler 910, particle modeler 912, database 914, and bitcomparator 916.

The system 900 may be operated in accordance with the processesdescribed in FIG. 10.

Embodiments of the techniques described in conjunction with FIG. 1-FIG.5 may be utilized in a process 1000 for interacting with substances.Referring to FIG. 10, in block 1002, the process 1000 receives sensorinput from a plurality of sensors via a multiplexer. In block 1004, theprocess 1000 associates the inputs with a substance.

In block 1006, the process 1000 analyzes the sensor input with an imageprocessor to extract data about the substance using optical flow andobject recognition and in block 1008 calibrates a vector modeler.

In block 1010, the process 1000 compares the multiple predictions totesting outcomes with a bit comparator, configures a processor to applya best fit analysis to eliminate non-matching possibilities, andestablishes physical property values to estimate substance composition.

In block 1012, the process 1000 creates a data model of the substance bycompiling the physical property values. In block 1014, process 1000queries a database for matching substances with matching physicalproperties to those of the substance.

In block 1016, process 1000 selects an action to perform on thesubstance with an action multiplexer. In block 1018, the action isapplied to the substance to generate a predictive model of a resultingaction through actuation of the particle modeler. In done block 1020,the process 1000 ends.

By predicting the interactions of substances, the process 1000 allowsfor the efficient control of peripheral devices and machines. Byanalyzing substances and interactions, the process 1000 providespredictive and real-time feedback about the substances and the actionsperformed for purposes of recording and repeating material handlingoutcomes.

Thus in some embodiments, an augmented reality assisted method ofidentifying interactions of substances involves receiving sensor inputfrom a group of sensors via a multiplexer, applying a substance analysisto one or more substances in an ocular field of view, selecting anaction with an action multiplexer, applying the action to thesubstance(s) and generating a predictive model of an interaction resultthrough actuation of a particle modeler.

The sensors used to gather information about the substances and theenvironment may include a user interface that may be operable by a humanoperator 106 to input observational data directly into an augmentedreality measurement system. Inputs may include verbal, typed, orhandwritten written descriptions about such things as the nature,composition, or state of a substance. The user interface and sensors mayinclude a microphone and speech recognition logic to process narrativeinput from the user into observational data. The sensors may furtherinclude one or more of a camera, mass spectrometer, thermal sensor, andpressure sensor.

The augmented reality measurement system may query data sources such asequipment and instrument database 114 and sample database 120, searchengines, or a reference library 606 that includes digital materialsrelated to physical chemistry, chemistry, microbiology, physics,material handling for a particular task (e.g., cooking), mathematics,and/or previously recorded procedural logs. In some case the sensors maybe operated in conjunction with optical character recognition (OCR)logic to process written notes of the human operator 106 intoobservational data.

The sensor input may be applied to an image processor to extract dataabout one or more substances using optical flow and object recognition.Predicting physical properties of the substance may be accomplished bycalibrating a vector modeler and a particle modeler to generate multiplepredictions about particle movement within the substance. Thesepredictions may be based on applied stored data models for the substancephysical properties. Predictions may include substance transformationsand state changes, for example predictions based on the passage of timeand actions performed with or to the substance(s).

Multiple predictions from the system may be compared to stored epiricaloutcomes with a bit comparator, and a processor may be configured toapply a best fit analysis to eliminate non-matching possibilities andestablish physical property values to estimate substance composition. Adata model of the substance may be generated by compiling the predictedphysical property values, and/or correlating the predicted propertieswith stored properties of known substances.

The substance analysis and the action may be applied to a group ofsubstances and the particle modeler generates a predictive model.Generating the predictive model may include determining velocity andacceleration vectors for the substance.

FIG. 11 illustrates an embodiment of a system 1100 for guiding anddocumenting procedural actions of an action set through an augmentedreality device. The system 1100 comprises a user space 1102, aninteraction environment 1108, an augmented reality device 1122, anetworked repository 1146, and a distribution server 1148.

The augmented reality device 1122 comprises augmented reality device1122, image processing logic 1116, action set documentation logic 1152,behavior identification logic 1158, action set database 1160, objecttargeting logic 1118, overlay synchronization logic 1120, an augmentedreality physics engine 1144, a processor 1110, a plurality of internaltransducers 1104, a plurality of external transducers 1106, a visualdisplay 1126, and a communications link to the networked repository 1146and the distribution server 1148.

The plurality of internal transducers 1104 typically will include gazetracking devices, depth (distance) measurement devices, one or moremicrophone, one or more speaker, and other sensors and transducerscommonly known in the art.

The memory 1112 comprises a data set allocation of memory 1156 and anaction log allocation of memory 1154. The action log allocation ofmemory 1154 comprises interaction entries 1162. The plurality ofexternal transducers 1106 identifies an environmental signal 1124.

The processor 1110 generates series of sequential image maps 1138 fromthe environmental signal 1124. The augmented reality physics engine 1144comprises an object recognition logic 1114 and access to a materialsdatabase 1136.

The visual display 1126 comprises an ocular field of view 1142 directedtowards a graphical overlay 1140. The graphical overlay 1140 comprises agraphical identifier 1150.

The interaction environment 1108 comprises an environmental objects1128. The environmental objects 1128 comprises an equipment andinstruments 1132 and a materials and products 1130. The system 1100 maybe operated in accordance with the process described in FIG. 12.

Referring to FIG. 12, in block 1202, the procedure 1200 provides theaugmented reality device, operatively disposed to a user space. In block1204, the procedure 1200 provides a graphical overlay orthogonallypositioned to an ocular field of view on the visual display. In block1206, the procedure 1200 transforms an environmental signal, capturedthrough the plurality of external transducers, into an image map.

In block 1208, the procedure 1200 recognizes equipment and instrumentsfrom the detected objects through comparative analysis of visualcharacteristics of the detected objects with known visualcharacteristics in an equipment and instrument database. In block 1210,the procedure 1200 recognizes materials and products from the detectedobjects through spatiotemporal movement analysis, comparative physicalmodeling, and referencing a materials database.

In block 1212, the procedure 1200 synchronizes the graphical overlay tocoincidently align the detected objects of the image map with theenvironmental objects of the interaction environment to the ocular fieldof view through operations of the processor controlled by overlaysynchronization logic.

In block 1214, the procedure 1200 recognizes objects of interest withinthe image map by transforming an ocular movement tracking signal,captured by the plurality of internal transducers, into an ocular lineof sight directed towards an environmental object in the interactionenvironment, with a corresponding detected object in the image map.

In block 1216, the procedure 1200 recognizes procedural action throughcontextual visual behavior recognition by tracking spatiotemporalmovement of the materials and products relative to the equipment andinstruments in the series of sequential image maps and correlatinginteractions to documented procedural actions in an action set database.

In block 1218, the procedure 1200 captures quantifiable data inquantifying features of the equipment and instruments through contextualfeature analysis by correlating the procedural action.

In block 1220, the procedure 1200 visualizes a guided interactionoverlay as the graphical overlay in response to an action set selectioninput of a documented action set from the action set database. In doneblock 1222, the procedure 1200 ends.

FIG. 13 is an example block diagram of a computing device 1300 that mayincorporate embodiments of the present invention, e.g. computer system116 (e.g., a desktop computer, server computer, smart phone, tablet)and/or augmented reality headset 108/augmented reality device 1122. FIG.13 is merely illustrative of a machine to carry out aspects of thetechnical processes described herein, and does not limit the scope ofthe claims. One of ordinary skill in the art would recognize othervariations, modifications, and alternatives.

In one embodiment, the computing device 1300 typically includes amonitor or graphical user interface 1302, a data processing system 1320,a communication network interface 1312, input device(s) 1308, outputdevice(s) 1306, and the like.

As depicted in FIG. 13, the data processing system 1320 may include oneor more processor(s) 1304 that communicate with a number of peripheraldevices via a bus subsystem 1318. These peripheral devices may includeinput device(s) 1308, output device(s) 1306, communication networkinterface 1312, and a storage subsystem, such as a volatile memory 1310and a nonvolatile memory 1314.

The volatile memory 1310 and/or the nonvolatile memory 1314 may storecomputer-executable instructions and thus forming logic 1322 that whenapplied to and executed by the processor(s) 1304 implement embodimentsof the processes disclosed herein.

The input device(s) 1308 include devices and mechanisms for inputtinginformation to the data processing system 1320. These may include akeyboard, a keypad, a touch screen incorporated into the monitor orgraphical user interface 1302, audio input devices such as voicerecognition systems, microphones, and other types of input devices. Invarious embodiments, the input device(s) 1308 may be embodied as acomputer mouse, a trackball, a track pad, a joystick, wireless remote,drawing tablet, voice command system, eye tracking system, and the like.The input device(s) 1308 typically allow a user to select objects,icons, control areas, text and the like that appear on the monitor orgraphical user interface 1302 via a command such as a click of a buttonor the like.

The output device(s) 1306 include devices and mechanisms for outputtinginformation from the data processing system 1320. These may includespeakers, printers, infrared LEDs, and so on as well understood in theart.

The communication network interface 1312 provides an interface tocommunication networks (e.g., communication network 1316) and devicesexternal to the data processing system 1320. The communication networkinterface 1312 may serve as an interface for receiving data from andtransmitting data to other systems. Embodiments of the communicationnetwork interface 1312 may include an Ethernet interface, a modem(telephone, satellite, cable, ISDN), (asynchronous) digital subscriberline (DSL), FireWire, USB, a wireless communication interface such asBlueTooth or WiFi, a near field communication wireless interface, acellular interface, and the like.

The communication network interface 1312 may be coupled to thecommunication network 1316 via an antenna, a cable, or the like. In someembodiments, the communication network interface 1312 may be physicallyintegrated on a circuit board of the data processing system 1320, or insome cases may be implemented in software or firmware, such as “softmodems”, or the like.

The computing device 1300 may include logic that enables communicationsover a network using protocols such as HTTP, TCP/IP, RTP/RTSP, IPX, UDPand the like.

The volatile memory 1310 and the nonvolatile memory 1314 are examples oftangible media configured to store computer readable data andinstructions to implement various embodiments of the processes describedherein. Other types of tangible media include removable memory (e.g.,pluggable USB memory devices, mobile device SIM cards), optical storagemedia such as CD-ROMS, DVDs, semiconductor memories such as flashmemories, non-transitory read-only-memories (ROMS), battery-backedvolatile memories, networked storage devices, and the like. The volatilememory 1310 and the nonvolatile memory 1314 may be configured to storethe basic programming and data constructs that provide the functionalityof the disclosed processes and other embodiments thereof that fallwithin the scope of the present invention.

Software that implements embodiments of the present invention may bestored in the volatile memory 1310 and/or the nonvolatile memory 1314.Said software may be read from the volatile memory 1310 and/ornonvolatile memory 1314 and executed by the processor(s) 1304. Thevolatile memory 1310 and the nonvolatile memory 1314 may also provide arepository for storing data used by the software.

The volatile memory 1310 and the nonvolatile memory 1314 may include anumber of memories including a main random access memory (RAM) forstorage of instructions and data during program execution and a readonly memory (ROM) in which read-only non-transitory instructions arestored. The volatile memory 1310 and the nonvolatile memory 1314 mayinclude a file storage subsystem providing persistent (non-volatile)storage for program and data files. The volatile memory 1310 and thenonvolatile memory 1314 may include removable storage systems, such asremovable flash memory.

The bus subsystem 1318 provides a mechanism for enabling the variouscomponents and subsystems of data processing system 1320 communicatewith each other as intended. Although the communication networkinterface 1312 is depicted schematically as a single bus, someembodiments of the bus subsystem 1318 may utilize multiple distinctbusses.

It will be readily apparent to one of ordinary skill in the art that thecomputing device 1300 may be a mobile device such as a smartphone, adesktop computer, a laptop computer, a rack-mounted computer system, acomputer server, or a tablet computer device. As commonly known in theart, the computing device 1300 may be implemented as a collection ofmultiple networked computing devices. Further, the computing device 1300will typically include operating system logic (not illustrated) thetypes and nature of which are well known in the art.

Those having skill in the art will appreciate that there are variouslogic implementations by which processes and/or systems described hereincan be effected (e.g., hardware, software, or firmware), and that thepreferred vehicle will vary with the context in which the processes aredeployed. If an implementer determines that speed and accuracy areparamount, the implementer may opt for a hardware or firmwareimplementation; alternatively, if flexibility is paramount, theimplementer may opt for a solely software implementation; or, yet againalternatively, the implementer may opt for some combination of hardware,software, or firmware. Hence, there are numerous possibleimplementations by which the processes described herein may be effected,none of which is inherently superior to the other in that any vehicle tobe utilized is a choice dependent upon the context in which theimplementation will be deployed and the specific concerns (e.g., speed,flexibility, or predictability) of the implementer, any of which mayvary. Those skilled in the art will recognize that optical aspects ofimplementations may involve optically-oriented hardware, software, andor firmware.

Those skilled in the art will appreciate that logic may be distributedthroughout one or more devices, and/or may be comprised of combinationsmemory, media, processing circuits and controllers, other circuits, andso on. Therefore, in the interest of clarity and correctness logic maynot always be distinctly illustrated in drawings of devices and systems,although it is inherently present therein. The techniques and proceduresdescribed herein may be implemented via logic distributed in one or morecomputing devices. The particular distribution and choice of logic willvary according to implementation.

The foregoing detailed description has set forth various embodiments ofthe devices or processes via the use of block diagrams, flowcharts, orexamples. Insofar as such block diagrams, flowcharts, or examplescontain one or more functions or operations, it will be understood asnotorious by those within the art that each function or operation withinsuch block diagrams, flowcharts, or examples can be implemented,individually or collectively, by a wide range of hardware, software,firmware, or virtually any combination thereof. Portions of the subjectmatter described herein may be implemented via Application SpecificIntegrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs),digital signal processors (DSPs), or other integrated formats. However,those skilled in the art will recognize that some aspects of theembodiments disclosed herein, in whole or in part, can be equivalentlyimplemented in standard integrated circuits, as one or more computerprograms running on one or more processing devices (e.g., as one or moreprograms running on one or more computer systems), as one or moreprograms running on one or more processors (e.g., as one or moreprograms running on one or more microprocessors), as firmware, or asvirtually any combination thereof, and that designing the circuitry orwriting the code for the software or firmware would be well within theskill of one of skill in the art in light of this disclosure. Inaddition, those skilled in the art will appreciate that the mechanismsof the subject matter described herein are capable of being distributedas a program product in a variety of forms, and that an illustrativeembodiment of the subject matter described herein applies equallyregardless of the particular type of signal bearing media used toactually carry out the distribution. Examples of a signal bearing mediainclude, but are not limited to, the following: recordable type mediasuch as floppy disks, hard disk drives, CD ROMs, digital tape, flashdrives, SD cards, solid state fixed or removable storage, and computermemory.

In a general sense, those skilled in the art will recognize that thevarious aspects described herein which can be implemented, individuallyor collectively, by a wide range of hardware, software, firmware, or anycombination thereof can be viewed as being composed of various types ofcircuitry.

Those skilled in the art will recognize that it is common within the artto describe devices or processes in the fashion set forth herein, andthereafter use standard engineering practices to integrate suchdescribed devices or processes into larger systems. At least a portionof the devices or processes described herein can be integrated into anetwork processing system via a reasonable amount of experimentation.Various embodiments are described herein and presented by way of exampleand not limitation.

What is claimed is:
 1. A method of guiding and documenting procedures ofan action set through an augmented reality device, the methodcomprising: providing the augmented reality device, operatively disposedto a user space, comprising a processor, memory, a plurality of externaltransducers, a plurality of internal transducers, and a visual display;generating a graphical overlay to an ocular field of view on the visualdisplay; transforming an environmental signal, captured through one orboth of the plurality of external transducers and the plurality ofinternal transducers, into an image map comprising detected objects,wherein the processor executes image processing logic, stored in thememory, to detect environmental objects within an interactionenvironment from the environmental signal; recognizing materials andproducts from the detected objects through spatiotemporal movementanalysis, comparative physical modeling, and referencing a materialsdatabase, wherein the processor is controlled by object recognitionlogic, stored in the memory, to recognize the materials and productsfrom the detected objects in a series of sequential image maps;synchronizing the graphical overlay to coincidentally align the detectedobjects of the image map with the environmental objects of theinteraction environment to the ocular field of view through operationsof the processor controlled by overlay synchronization logic;recognizing procedural actions through contextual visual behaviorrecognition by tracking spatiotemporal movement of the materials andproducts relative to the materials and products in the series ofsequential image maps and correlating to documented procedural actionsin an action set database as controlled by behavior identificationlogic; capturing quantifiable data in quantifying features of thematerials and products through contextual feature analysis bycorrelating the procedural actions and an ocular line of sight to thequantifying features of the materials and products in the series ofsequential image maps; visualizing a guided interaction overlay as thegraphical overlay in response to a selection input of a documentedaction set from the action set database; identifying upcoming actionsfrom the documented action set and corresponding equipment andinstrument identifiers for the upcoming actions; identifying equipmentand instruments in the graphical overlay representing the correspondingequipment and instrument identifiers; warming the correspondingequipment and instrument identifiers in the graphical overlay inproportion to their distance from a current position in the documentedaction set; and providing on the augmented reality device a comparisonof the action set as carried out by a user and an ideal version of theaction set, in response to the user stopping motion and requesting theideal version of the action set via the augmented reality device.
 2. Themethod of claim 1 further comprising: visualizing an action set outlinedetailing the documented procedural actions of the documented action setthrough the guided interaction overlay; visualizing a list of materials,and a list of the equipment and the instruments, utilized in thedocumented action set through the guided interaction overlay; displayinga graphical identifier coincidentally aligned to the detected objects inthe image map corresponding to the materials, the equipment, and theinstruments in the interaction environment through the guidedinteraction overlay; and displaying advisement notifications through theguided interaction overlay in response to detecting an advisement andprecaution alert associated with a particular documented proceduralaction.
 3. The method of claim 2, wherein the list of materials and thelist of equipment and instruments are visualized through the guidedinteraction overlay for each procedure of the action set.
 4. The methodof claim 1, wherein the graphical overlay is positioned between theocular field of view and a graphical visualization of the interactionenvironment.
 5. The method of claim 2, further comprising: receiving auser interaction via the augmented reality device to modify a storedrepresentation of the action set outline.
 6. The method of claim 1,wherein the comparison comprises a comparison of the actual behavior ofpowders and liquids in the user space and an ideal behavior of thepowders and liquids as defined in the action set.
 7. The method of claim1, further comprising: exchanging a field of view of the user space froma first user to a second user via the augmented reality device.
 8. Themethod of claim 1, further comprising: a user pouring liquid from acontainer to generate droplets; providing feedback to the user via theaugmented reality device to modify a speed of generating the droplets.9. The method of claim 8, the feedback comprising: displaying on theaugmented reality device an angle of orientation for the container. 10.The method of claim 1, further comprising: calibrating user force on acontainer via a force feedback sensor; and applying the calibration todetermine a user-initiated disturbance to material in the container. 11.The method of claim 1, further comprising: determining characteristicsof powdered or granular materials based on an angle of repose of acontainer of the powdered or granular materials as detected via theaugmented reality device.
 12. The method of claim 11, furthercomprising: receiving hand gestures of a user via the augmented realitydevice; and applying the hand gestures to identify boundaries of thepowdered or granular materials.
 13. The method of claim 1, furthercomprising: warming the corresponding equipment and instrumentidentifiers in the graphical overlay in proportion to their temporalrelevance from a current position in the documented action set.